Security Features-Qualcomm OTA
Introduction
Qualcomm Linux supports Over-the-Air (OTA) updates through a combination of UEFI Capsule Updates and OSTree. UEFI provides a standardized framework for firmware initialization and management, including a secure mechanism known as Capsule Updates for updating firmware images.[1] A UEFI capsule packages firmware images together with metadata and authentication information, allowing the platform firmware to verify and safely apply updates during the boot process.[2]
For OS updates, Qualcomm Linux uses OSTree,a filesystem deployment and update framework. OSTree stores snapshots of the OS in a temporary repository and enables atomic upgrades and rollback functionality, ensuring that devices can recover safely from interrupted or failed updates. Together, UEFI Capsule Updates and OSTree provide a secure and reliable OTA solution, where firmware and operating system components can be updated independently while maintaining system integrity and rollback capabilities.
To update firmware and OS in a single OTA system, the capsule update handles first the firmware updates within the low-level firmware. Once that is done, the system reboots and reaches Linux OS, where it checks for and applies the OSTree updates. The capsule update sequence is as follows:
- A binary known as a UEFI capsule encapsulates the firmware update.
- The system delivers the capsule binary to the UEFI by storing it in the mounted
/EFIpath. - The UEFI firmware processes the capsule during the boot cycle, and applies the update to the device firmware.
Regarding the OSTree, the system copies the Qualcomm Linux build-generated update package to the device and stages it for activation. After the board reboots, the update will be applied. These are the Linux OS and firmware images that are updated as part of OTA:
- Linux OS images:
efi.binfile which contains the UKI, intrd, and boot loader configuration files. OSTree creates a new configuration file during deployment. It also contains a list of the paths to the new kernel and initramfs images copied to the EFI partition.system.imgfile which contains the rootfs, including/ostree,/ostree/repo, and/ostree/deploy. The OSTree updates the filesystem tree each time a new deployment is created, reflecting the new version of the OS.
- Firmware images:
- You can find the full list of firmware images and their supported targets in this link.
Update capsule and HLOS
To update the capsule and High Level Operating System (HLOS) follow these instructions:
1. Copy the <capsule>.cap capsule file to EFI partition, which is mounted at /boot/EFI/UpdateCapsule on the booted-up device.
2. Set the EFI variable (efivar) OsIndications flags with EFI_OS_INDICATIONS_FILE_CAPSULE_DELIVERY_SUPPORTED and reboot the device. UEFI performs the following steps when it detects a capsule update request through the OsIndications flag:
- 2.1. UEFI identifies that there is a capsule available for update from
OsIndicationsflag.
- 2.2. UEFI authenticates the capsule and updates the firmware images from the capsule.
- 2.3. The status of capsule update is updated in the EFI system resource table (ESRT).
- 2.4. If there is a failure during capsule update, UEFI rolls back the firmware to the previous version.
3. Copy the OSTree repository to the device. Create a new deployment for HLOS update using OSTree commands, a new configuration file with count tag gets created, reboot the device.
4. Systemd-boot picks the new configuration file and boots up the kernel and user space. The device boots up with the updated firmware and the HLOS software. systemd-bless-boot.service marks the new configuration as good.
5. Reset the TrialBootEnabled flag in OtaStatus efivar to indicate there is no issue with the deployed firmware. UEFI checks this efivar to commit the new firmware.
Create UEFI Capsule
Prerequisites:
- OpenSSL
- Python3
- Git
- Capsule Generation Tools Repo (Find installation instructions here.)
Capsule Certificates
Before starting the capsule generation process, OpenSSL certificates are required. Follow these steps to create the certificates that will be applied to the update capsule:
1. Install dependencies:
sudo apt update sudo apt install openssl python3 git
2. Create a directory to store certificates:
export CERTIFICATES_DIR=<your-desired-certificates-path> mkdir -p $CERTIFICATES_DIR
3. Create the Root Certificate:
openssl genrsa -out QcFMPRoot.key 2048 openssl req -new -x509 -days 3650 -key QcFMPRoot.key -out QcFMPRoot.crt -subj "/C=<COUNTRY>/ST=<STATE>/L=<LOCALITY>/O=<ORGANIZATION>/CN=<ROOT_CA_COMMON_NAME>" openssl x509 -in QcFMPRoot.crt -out QcFMPRoot.cer -outform DER openssl x509 -inform DER -in QcFMPRoot.cer -outform PEM -out QcFMPRoot.pub.pem
4. Create the Intermediate Certificate
openssl genrsa -out QcFMPSub.key 2048 openssl req -new -key QcFMPSub.key -out QcFMPSub.csr -subj "/C=<COUNTRY>/ST=<STATE>/L=<LOCALITY>/O=<ORGANIZATION>/CN=<SUB_CA_COMMON_NAME>" openssl x509 -req -in QcFMPSub.csr -CA QcFMPRoot.crt -CAkey QcFMPRoot.key -CAcreateserial -days 3650 -out QcFMPSub.crt -sha256 openssl x509 -in QcFMPSub.crt -out QcFMPSub.cer -outform DER openssl x509 -inform DER -in QcFMPSub.cer -outform PEM -out QcFMPSub.pub.pem
5. Create the Capsule Signing Certificate
openssl genrsa -out QcFMPCert.key 2048 openssl req -new -key QcFMPCert.key -out QcFMPCert.csr -subj "/C=<COUNTRY>/ST=<STATE>/L=<LOCALITY>/O=<ORGANIZATION>/CN=<SIGNING_CERT_COMMON_NAME>" openssl x509 -req -in QcFMPCert.csr -CA QcFMPSub.crt -CAkey QcFMPSub.key -CAcreateserial -days 3650 -out QcFMPCert.crt -sha256 cat QcFMPCert.key QcFMPCert.crt > QcFMPCert.pem
Capsule Generation Tools
Generate the update capsule following these steps:
1. Create a directory for the Capsule Generation Tools, and install it:
export CAPSULE_TOOL_DIR=<your-desired-path> mkdir -p $CAPSULE_TOOL_DIR cd $CAPSULE_TOOL_DIR git clone https://github.com/quic/cbsp-boot-utilities.git cd cbsp-boot-utilities/uefi_capsule_generation pipx install . qcom-capsule-tool setup
2. Create a directory for the encapsulated images:
export ENCAPSULATED_IMAGES=<your-desired-path> mkdir -p $ENCAPSULATED_IMAGES
3. Copy the desired images to update into the previous directory. Usually this images will be at the deploy directory of the Yocto build. The path should be similar to $BSP-PATH/build-<image>/tmp-glibc/deploy/images/qcs9075-iq-9075-evk/$IMAGE/. For this guide, the xbl.elf, xbl_config.elf, uefi.elf and tz.mbn images will be updated as an example but choose as needed the desired images:
cp /path/to/xbl.elf $ENCAPSULATED_IMAGES/ cp /path/to/xbl_config.elf $ENCAPSULATED_IMAGES/ cp /path/to/uefi.elf $ENCAPSULATED_IMAGES/ cp /path/to/tz.mbn $ENCAPSULATED_IMAGES/
4. Generate the firmware version:
qcom-capsule-tool sysfw-version-create -Gen -FwVer 0.0.A.B -LFwVer 0.0.0.0 -O SYSFW_VERSION.bin
Where A and B are the version numbers, and these are options available:
-Gen: Generates a new firmware version file.-FwVer: Specifies the firmware version.-LFwVer: Specifies the lowest firmware version.-O: Output file name.
4.1 To print the Firmware Versions in the .bin file:
qcom-capsule-tool sysfw-version-create --PrintAll SYSFW_VERSION.bin
5. Generate the Firmware XML for the Dragonwing IQ-9075 chipset + UFS:
qcom-capsule-tool update-fv-xml -S UFS -T QCS9100
6. Modify the FvUpdate.xml, and update the Operation field for each firmware entry as needed. By default the operation is set to IGNORE. For the example images, the change would be as follows (modify the commands as needed):
sed -i '/<InputBinary>xbl\.elf<\/InputBinary>/,/<\/FwEntry>/s/<Operation>IGNORE<\/Operation>/<Operation>UPDATE<\/Operation>/' FvUpdate.xml sed -i '/<InputBinary>xbl_config\.elf<\/InputBinary>/,/<\/FwEntry>/s/<Operation>IGNORE<\/Operation>/<Operation>UPDATE<\/Operation>/' FvUpdate.xml sed -i '/<InputBinary>uefi\.elf<\/InputBinary>/,/<\/FwEntry>/s/<Operation>IGNORE<\/Operation>/<Operation>UPDATE<\/Operation>/' FvUpdate.xml sed -i '/<InputBinary>tz\.mbn<\/InputBinary>/,/<\/FwEntry>/s/<Operation>IGNORE<\/Operation>/<Operation>UPDATE<\/Operation>/' FvUpdate.xml
7. Create the firmware volume:
qcom-capsule-tool fv-create firmware.fv -FvType SYS_FW FvUpdate.xml SYSFW_VERSION.bin $ENCAPSULATED_IMAGES/
8. Update the JSON Parameters:
qcom-capsule-tool update-json -j config.json -f SYS_FW -b SYSFW_VERSION.bin -pf firmware.fv -p $CERTIFICATES_DIR/QcFMPCert.pem -x $CERTIFICATES_DIR/QcFMPRoot.pub.pem -oc $CERTIFICATES_DIR/QcFMPSub.pub.pem -g 78462415-6133-431C-9FAE-48F2BAFD5C71
9. Generate the Capsule File:
PYTHONPATH=src/qcom_capsule_tool/edk2/BaseTools/Source/Python python3 src/qcom_capsule_tool/edk2/BaseTools/Source/Python/Capsule/GenerateCapsule.py -e -j config.json -o <firmware_capsule>.cap --capflag PersistAcrossReset -v
Update firmware using capsule
To update the firmware using a capsule file, follow these steps on the target board:
1. Create the UpdateCapsule directory on the target board:
mkdir -p /boot/EFI/UpdateCapsule
2. Copy the capsule to the device from the host machine:
scp -r <firmware_capsule>.cap <user>@<IP_address>:/boot/EFI/UpdateCapsule/<firmware_capsule>.cap
3. Create the data.hex file on the device with the specified hexadecimal data:
echo -e -n "\x4\x0\x0\x0\x0\x0\x0\x0" > data.hex
4. Write the contents of data.hex on the device to the UEFI variable OsIndications using the efivar tool:
efivar -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-OsIndications -f data.hex -w
5. Print the value of the OsIndications UEFI variable using the efivar tool:
efivar -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-OsIndications -p
You should see a similar output to this:
GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
Name: "OsIndications"
Attributes:
Non-Volatile
Boot Service Access
Runtime Service Access
Value:
00000000 04 00 00 00 00 00 00 00 |........ |
6. Reboot the device:
reboot
7. Check the ESRT table entries:
cd /sys/firmware/efi/esrt/entries/entry0
- 7.1 Check the output of
last_attempt_statuscommand. If its value is 0, it means that the update was successful:
cat last_attempt_status
- 7.2. Check the output of
last_attempt_version:
cat last_attempt_version
- 7.3. Check the output of
fw_version. Iflast_attempt_versionandfw_versionhave the same output, the update was successful:
cat fw_version
Update Linux OS using OSTree
To update Linux OS using OSTree, follow these steps:
1. Check the current deployment in the board:
ostree admin status
Expected output:
* poky d8bc01a4fd76550aef9668e11d33163f482bee24f15e9201892934a5ac1ca063.0
Version: 1.8-ver.1.1
origin refspec: poky:qcs9075-iq-9075-evk
The * symbol is an indicator for the current deployment that the device has booted with.
2. The ostree_repo package is in the $WORK_DIR/build-<image>/tmp-glibc/deploy/images/qcs9075-iq-9075-evk/ path on the host development computer. Copy the ostree_repo package from the host computer to the Dragonwing IQ-9075:
scp -r ostree_repo <user>@<IP_address>:/tmp
3. Pull a local OSTree repository on the Dragonwing IQ-9075:
ostree pull-local /tmp/ostree_repo <branch_name>
For the Dragonwing IQ-9075, the command should look like:
ostree pull-local /tmp/ostree_repo qcs9075-iq-9075-evk
To find the branch_name needed, run on the device:
ostree refs
Expected output:
poky:qcs9075-iq-9075-evk
The preceding output should display a similar branch name, referencing the Dragonwing IQ-9075 board.
4. Create the deployment on the Qualcomm device:
ostree admin deploy <branch_name>
This creates the ostree-2-poky.conf configuration file in the /boot/loader/entries directory. For the Dragonwing IQ-9075, the command should look like:
ostree admin deploy qcs9075-iq-9075-evk
5. Reboot the device:
reboot
6. Check if the device is booted with the created deployment:
ostree admin status
* poky d8bc01a4fd76550aef9668e11d33163f482bee24f15e9201892934a5ac1ca063.1
Version: 1.8-ver.1.1
origin refspec: qcs9075-iq-9075-evk
poky d8bc01a4fd76550aef9668e11d33163f482bee24f15e9201892934a5ac1ca063.0 (rollback)
Version: 1.8-ver.1.1
origin refspec: poky:qcs9075-iq-9075-evk
The rollback option appears as the system was updated with the same image already installed. Modify the source code as needed to change the image to be deployed. To verify the newly created deployment on the build host computer, check the <workspace>/build-<image>/tmp-glibc/work/iq-9075-evk/<IMAGE>/ota-sysroot/ostree/deploy/poky/deploy path.
Rollback system
A rollback system is a mechanism that allows a device to automatically return to a previously working software version when a newly installed update fails to boot or operate correctly. This capability is commonly required in embedded and IoT devices, where physical access to recover a failed system may be limited or unavailable. Rollback functionality improves system reliability by ensuring that software updates do not leave the device in an unusable state.
OSTree supports rollback by preserving previous filesystem deployments that can be selected during boot. On systems using systemd-boot, the boot counting mechanism is one of the components that can be used to implement rollback functionality. It tracks the number of unsuccessful boot attempts after an update and can automatically revert to a previously working deployment when a configurable threshold is exceeded.
Successful boot
To manage boot counting in systemd-boot after a successful boot, OSTree executes the following procedure:
1. As a new configuration is deployed by OSTree, it creates a configuration file with a +3 tag in the name, indicating the maximum retry count. This allows boot counting.
2. systemd-boot detects the numerical tag in the entry file name and renames it to ostree-conf+2-1.conf, indicating that one boot attempt has started. After renaming the file, the boot process continues.
3. The systemd-bless-boot-generator creates the systemd-bless-boot.service, which is set to start when boot-complete.target is reached.
4. The systemd-bless-boot.service marks the new configuration as successful by removing the counter tags +2-1 and renaming the file to /ostree-conf.conf.
Unsuccessful boot and rollback
To manage boot counting in systemd-boot after an unsuccessful boot and rollback flow:
1. As a new configuration is deployed by OSTree, it creates a configuration file with a +3 tag in the name, indicating the maximum retry count. This allows boot counting.
2. systemd-boot detects the numerical tag in the entry file name and renames it to ostree-conf+2-1.conf, indicating that one boot attempt has started. After renaming the file, the boot process continues.
3. The systemd-bless-boot-generator creates the systemd-bless-boot.service, which is set to start when boot-complete.target is reached. If there are any failures during the Linux boot-up, the systemd-bless-boot.service does not remove the +2-1 counter tags from the configuration file.
4. On the subsequent boot, systemd-boot detects the +2-1 tag in the configuration file name, renames the file to ostree-boot+1-2.conf, and tries to boot with it.
5. If the Linux bootup fails on the second attempt, the systemd-bless-boot.service does not remove the counter tags +1-2 from the configuration file.
6. On the next boot, systemd-boot detects the +1-2 tag in the configuration file name, renames the file to ostree-boot+0-3.conf, and tries to boot with it. This is the last attempt to boot Linux deployment.
7. If the device fails to boot Linux during the third attempt, the systemd-bless-boot.service does not remove the counter tags +0-3 from the configuration file.
8. On the subsequent boot, systemd-boot finds the +0-3 tag in the configuration file name. As the counter has reached zero, the entry configuration file is considered corrupt. The systemd-boot reverts to an earlier version by trying the valid configuration file entry.
Usrmerge
Usrmerge is a Linux filesystem reorganization approach that consolidates traditional system directories under the /usr hierarchy. Instead of maintaining separate locations such as /bin, /sbin, and /lib, their contents are stored in /usr/bin, /usr/sbin, and /usr/lib. The original directories remain available as symbolic links, preserving compatibility with existing applications and scripts.
By centralizing binaries and libraries in a single location, Usrmerge simplifies system maintenance and reduces duplication within the filesystem. This unified layout eliminates the distinction between root-level and /usr directories for most executable files and libraries, while symbolic links ensure that software referencing legacy paths continues to operate without modification.
Many modern Linux distributions, including Debian, Ubuntu, and Fedora, have adopted Usrmerge as part of their default filesystem organization. This transition aligns with recommendations from the Filesystem Hierarchy Standard (FHS) and is particularly beneficial for containerized and embedded environments, where a simplified and more consistent root filesystem improves manageability and deployment.
Manage /var, /home,/media, /mnt, /opt, /srv, and /usr in Qualcomm Linux
OSTree considers /var as a persistent directory. This means user and runtime created contents under /var remain untouched by OSTree and persist across OTA updates. Other directories that remain untouched by OSTree during an OTA update are /var, /home,/media, /mnt, /opt,and /srv. OSTree maps these directories as symbolic links under /var/rootdirs/<path-name>, for example, /home is a symbolic link to /var/rootdirs/home. Any runtime data stored under these directories stays persistent across OTA updates.
To maintain a clean and consistent filesystem, don't install any artifacts under the preceding directories at build time. Any artifacts installed in these directories during build time are not packaged into the rootfs image generated by the Qualcomm Linux build command, bitbake <image-recipe>. To handle runtime creation of files and directories under persistent paths, do the following:
1. Paths under /run/,/var/lib, /var/cache, and /var/log/ can be created from the respective systemd unit files.
2. Use systemd-tmpfiles to create files, symbolic links, and directories at bootup.
OSTree creates a read only bind mount at /usr, ensuring that the core operating system files remain immutable by users. This approach helps maintain system integrity and security, OSTree uses the /usr mount-point to deploy the next update. It is important to note that OSTree allows installation of files and directories under the /var/local path at build time, and although OSTree preserves the contents installed under /usr during build time, it doesn't package the contents installed under the /usr/local subdirectory into the rootfs image.
In Qualcomm Linux with OSTree enabled, the /etc directory is managed in a way that allows for both system updates and local customizations.
- The
/etcdirectory is mutable, allowing modifications at runtime for maintaining system configurations that need to persist across updates. - When an update is applied, OSTree performs a three-way merge for configuration files in
/etcusing the original version of the file, updated version of the file and locally modified version of the file. - If conflicts arise during the merge process, OSTree retains the runtime modifications. This helps maintain system stability and ensures that critical configurations are not overwritten.
SOTA distribution feature
The software over-the-air (SOTA) distribution feature allows remote updates for embedded systems and IoT devices. It integrates tools such as OSTree for system updates, allowing devices to receive and install updates without physical access. To enable the SOTA distribution feature in Qualcomm Linux, run:
kas build meta-qcom/ci/rb3gen2-core-kit.yml:meta-qcom/ci/qcom-distro-sota.yml:meta-qcom/ci/linux-qcom-6.18.yml
This command automatically enables and configures the SOTA distribution feature for your build.
References